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January  25,  1993 

The  Honorable  Richard  B.  Cheney 
The  Secretary  of  Defense 


Dear  Mr.  Secretary: 


We  have  evaluated  the  Army’s  test  program  for  the  AVR-2  laser  warning 
system  and  a  newly  modified  version  designated  the  AVR-2A.  We  focused 
specifically  on  whether  the  testing  accomplished  justifies  the  systems’ 
full-rate  production  status. 


Background 


The  AVR-2  and  AVR-2A  are  intended  for  use  on  combat  helicopters  to 
protect  against  threat  weapons  that  rely  on  lasers  for  their  operation. 

(See  fig.  1.)  Some  of  these  weapons  are  guided  by  lasers  while  others  are 
aided  by  lasers  in  accomplishing  such  functions  as  determining  the  range 
to  target  aircraft.  The  AVR-2  and  AVR-2A  protect  against  such  threats  by 
detecting  laser  energy,  providing  an  audio  warning,  and  displaying  the  type 
and  location  of  the  threat  on  a  screen  in  the  helicopter  cockpit.  Based  on 
the  warning  information,  the  pilot  may  launch  an  attack  against  the  threat 
or  attempt  to  evade  it. 


The  Army  started  development  of  the  AVR-2  in  the  1970s.  After  the  system 
completed  initial  operational  tests  in  1985,  the  Army  modified  its  design  to 
overcome  performance  problems  and  awarded  the  initial  production 
contract  in  1988.  The  AVR-2A  subsequently  evolved  from  additional  design 
changes  to  the  AVR-2.  Full-rate  production  of  the  AVR-2  was  approved  in 
February  1992.  The  AVR-2A  is  expected  to  be  phased  into  production  in 
early  1993  and  deliveries  are  scheduled  to  start  in  July  1993.  So  far,  the 
Army  has  contracted  for  940  systems,  including  519  AVR-2s  and  421 
AVR-2As.  Total  program  requirements  are  for  1,782  systems  (1,472  for  the 
Army,  254  for  the  Marine  Corps,  and  56  for  the  Navy)  at  an  estimated 
program  cost  of  $261  million. 
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Figure  1:  Cobra  Attack  Helicopter  and 
AVR-2  Laser  Warning  System 
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Results  in  Brief 

Production  of  the  AVR-2  and  AVR-2A  should  be  minimized  until  successful 
completion  of  operational  tests.  The  AVR-2  failed  to  demonstrate 
acceptable  performance  in  its  operational  tests  and  was  redesigned.  Even 
though  half  of  the  total  quantity  required  has  been  procured,  neither  the 
redesigned  AVR-2  nor  the  further  redesigned  AVR-2A  has  been  subjected 
to  operational  tests  to  ensure  satisfactory  performance.  This  is  contrary  to 
Department  of  Defense  (dod)  acquisition  policy,  and  as  a  result,  the  Army 
risks  procuring  an  inventory  of  systems  that  could  prove  defective. 

AVR-2  and  AVR-2A 
Have  Not 
Demonstrated 
Acceptable 
Performance  in 
Operational  Tests 

The  performance  capability  of  the  redesigned  AVR-2  and  AVR-2A  now 
being  procured  is  unproven  because  neither  system  has  been  subjected  to 
operational  tests.  At  the  time  of  our  review,  the  Army  did  not  plan  to 
conduct  any  such  operational  tests,  contrary  to  dod  policy. 

Importance  of  Operational 
Testing 

Operational  testing  is  DOD’s  primary  means  of  evaluating  weapon  system 
performance  in  a  combat  representative  environment  It  can  help  reduce 
risks  in  acquisition  programs  by  identifying  defective  systems  before  they 
are  produced  and  by  verifying  the  correction  of  performance  deficiencies. 
Operational  testing  is  distinguished  from  developmental  testing,  which  is 
done  to  verify  that  technical  performance  specifications  are  met  and  to 
determine  whether  a  system  is  ready  for  operational  testing. 

dod’s  acquisition  policy  provides  that  operational  testing  shall  be 
structured  to  determine  the  operational  effectiveness  and  suitability1  of  a 
system  under  realistic  combat  conditions  and  to  determine  if  the  minimum 
acceptable  operational  performance  requirements  have  been  satisfied.  The 
policy  further  provides  that  a  system  may  not  enter  full-rate  production 
until  test  results  provide  reasonable  assurance  that  the  design  is 
operationally  acceptable.  It  also  provides  that  one  of  the  objectives  of  the 
production  phase  is  to  conduct  follow-on  operational  testing  to  confirm 
system  performance  and  verify  the  correction  of  deficiencies. 

Operational  effectiveness  refers  to  the  ability  of  a  system  to  accomplish  its  mission  in  the  planned 
operational  environment  Operational  suitability  is  the  degree  to  which  a  system  can  be  placed 
satisfactorily  in  field  use  considering  such  factors  as  reliability  and  maintainability. 

PafeS 


GAO/N8IAD-98-14  Electronic  Warfkre 


B-250665 


No  Operational  Tests 
Planned 


Substitute  Testing  Is 
Inadequate 


During  operational  tests  in  1985,  the  original  AVR-2  failed  to  demonstrate 
acceptable  performance.  Mjyor  shortcomings  included  an  inability  to 
detect  multiple  threats,  an  inability  to  operate  properly  with  the 
helicopter’s  APR-39A  radar  warning  receiver,2  and  a  tendency  to  provide 
false  warnings.  The  system  also  failed  to  demonstrate  that  it  met  reliability 
requirements.  In  addition,  the  system  configuration  tested  weighed 
22.2  pounds,  or  2.2  pounds  over  the  maximum  allowable  weight. 

Following  the  1985  operational  tests  of  the  AVR-2,  the  Army  made  design 
changes  to  improve  the  system’s  performance  and  reduce  its  weight. 

These  changes  included  redesigning  the  system’s  receiver  and  electronic 
components  at  a  cost  of  $3.4  million,  or  30  percent  of  the  system’s  original 
development  cost  of  $11.4  million.  The  changes  also  increased  unit 
production  cost  by  $3,890,  to  $125,251  each. 

In  early  1991,  the  Army  began  a  series  of  additional  design  changes  to  the 
AVR-2  to  improve  the  system’s  sensitivity  and  thereby  allow  detection  of 
certain  threat  systems  at  greater  ranges.  Modifications  were  also  made  to 
incorporate  a  device  that  will  allow  its  use  with  a  training  system  for  Army 
pilots.  The  modifications  cost  about  $8  million,  or  70  percent  of  the 
system’s  original  development  cost,  and  increased  unit  production  cost  by 
$9,061,  to  $134,312  each.  The  resulting  system  was  designated  the  AVR-2A. 

Despite  the  changes  in  system  design  and  dod’s  stated  policy,  the  Army 
decided  that  follow-on  operational  testing  was  not  necessary.  Consistent 
with  dod  policy,  the  Army  did  plan  to  conduct  follow-on  operational 
testing  in  1991  using  production  models  of  the  redesigned  AVR-2. 
According  to  the  test  plan,  several  critical  issues  were  to  be  evaluated, 
including  the  system’s  operational  effectiveness  and  reliability.  However, 
the  follow-on  test  was  canceled  because  of  Operation  Desert  Storm. 


Rather  than  conduct  follow-on  operational  testing,  the  Army  planned  to 
rely  on  technical  tests  to  demonstrate  system  performance.  However, 
these  tests  are  not  a  valid  substitute  for  operational  tests. 

Technical  testing  is  essentially  developmental  testing  and  is  not  done  in  a 
realistic  operational  environment  needed  to  evaluate  a  system’s 
operational  effectiveness  and  suitability.  For  example,  tests  to 
demonstrate  the  AVR-2’s  reliability  were  held  in  the  contractor’s  plant  and 
were  not  done  under  the  cognizance  of  the  Army’s  independent 


*A  radar  warning  receiver  is  an  electronic  warfare  system  to  want  of  radar  controlled  threat  systems. 
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operational  test  organization.  Moreover,  dod  policy  provides  for 
demonstrating  a  system’s  operational  effectiveness  and  suitability  in 
operational  tests,  not  technical  tests. 

Our  work  has  shown  that  failure  to  conduct  operational  testing  to  verify 
the  adequacy  of  redesigned  systems  can  result  in  adverse  consequences. 
On  a  recent  review  of  another  Army  electronic  warfare  system,3  we  found 
that  design  changes  were  made  to  correct  a  serious  shortcoming 
discovered  during  operational  testing.  However,  no  additional  operational 
testing  was  done  to  verify  the  adequacy  of  the  changes.  Subsequently, 
during  Operation  Desert  Storm,  the  system  proved  so  defective  that  Army 
pilots  stopped  using  it. 

In  addition,  at  the  time  of  our  review,  none  of  the  substitute  testing  had 
been  or  was  planned  to  be  done  on  the  Army’s  Apache  helicopter,  the 
main  aircraft  slated  to  use  the  AVR-2A.  During  a  recent  review  of  a  Navy 
electronic  warfare  system,  we  found  that  the  system  had  successfully 
passed  operational  testing  on  a  certain  helicopter.  Based  on  those  tests, 
the  Navy  procured  several  hundred  systems  and  then  operationally  tested 
the  system  on  another  helicopter  scheduled  to  use  the  system.  Those  tests 
showed  that  the  system  degraded  rather  than  enhanced  the  other 
helicopter’s  survivability. 

Another  limitation  of  the  Army’s  substitute  testing  is  illustrated  by  the 
AVR-2’s  electromagnetic  compatibility  tests.  These  tests  refer  to  the 
capability  of  systems  to  operate  in  their  intended  environments  without 
causing  or  suffering  from  interference  with  other  systems.  The  AVR-2 
electromagnetic  compatibility  tests  were  conducted  on  a  Scout  helicopter 
that  had  much  of  its  electronic  equipment  either  missing  or  inoperative.  In 
addition,  the  Scout  does  not  have  some  equipment,  such  as  the  ALQ-136 
radar  jammer,  that  is  used  on  the  Apache  and  Cobra  helicopters.  Thus,  the 
Army  has  little  assurance  of  the  AVR-2’s  electromagnetic  compatibility. 


Recommendation 


We  recommend  that  the  Secretary  of  Defense  require  that  production  of 
the  AVR-2  and  AVR-2A  be  minimized  until  operational  testing  provides 
reasonable  assurance  that  performance  is  satisfactory. 


Agency  Comments 


dod  partially  agreed  with  the  findings  in  this  report  but  disagreed  with  the 
recommendation,  dod  stated  that  adequate  operational  testing  had  been 


•This  was  reported  in  a  1992  classified  GAO  report 
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conducted  or  was  planned  to  reasonably  ensure  operationally  suitable  and 
effective  system  performance. 

However,  at  the  time  of  our  review,  operational  testing  of  the  AVR-2  had 
revealed  performance  deficiencies,  and  no  additional  operational  testing 
had  been  planned.  In  responding  to  the  draft  of  this  report,  dod  agreed  that 
operational  testing  on  the  Apache  helicopter  was  needed,  dod  officials 
informed  us  that  this  testing  was  planned  in  response  to  the  draft  report 
and  that  it  would  take  place  in  April  to  June  1993. 

Even  though  dod  disagreed  with  our  recommendation,  we  consider  dod’s 
planned  test  to  be  partially  responsive  to  our  recommendation,  assuming 
that  this  and  other  tests  indicated  in  dod’s  comments  are  accomplished  as 
planned.  We  believe  that  the  tests  will  reduce  the  Army’s  risk  of  procuring 
deficient  systems.  However,  we  also  believe  that  the  risk  could  be  further 
reduced  by  minimizing  production  until  the  tests  are  successfully 
accomplished. 

dod’s  detailed  comments  on  the  draft  of  this  report  are  reprinted  in 
appendix  I. 


Scope  and 
Methodology 


We  performed  our  work  at  the  Army’s  AVR-2  System  Program  Office, 

St  Louis,  Missouri;  the  Army  Communications-Electronics  Command,  Fort 
Monmouth,  New  Jersey;  the  Army  Operational  Test  and  Evaluation 
Command,  Alexandria,  Virginia;  and  the  Army  Aviation  Center  and  School, 
Fort  Rucker,  Alabama  We  reviewed  system  test  plans  and  reports, 
requirements  documents,  contractual  records,  system  configuration 
control  records,  dod  acquisition  policy  directives,  and  other  records 
bearing  on  the  issue.  We  also  discussed  various  aspects  of  our  work  with 
Army  officials.  We  performed  our  work  from  February  1992  through 
September  1992  in  accordance  with  generally  accepted  government 
auditing  standards. 


As  you  know,  31  U.S.C.  720  requires  the  head  of  a  federal  agency  to  submit 
a  written  statement  on  actions  taken  on  our  recommendation  to  the  House 
Committee  on  Government  Operations  and  the  Senate  Committee  on 
Governmental  Affairs  not  later  than  60  days  after  the  date  of  the  report.  A 
written  statement  must  also  be  submitted  to  the  House  and  Senate 
Committees  on  Appropriations  with  the  agency’s  first  request  for 
appropriations  made  more  than  60  days  after  the  date  of  the  report 
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We  are  sending  copies  of  this  report  to  the  Chairmen  of  the  above 
Committees  and  of  the  Senate  and  House  Committees  on  Armed  Services, 
the  Secretary  of  the  Army,  the  Director  of  the  Office  of  Management  and 
Budget,  and  other  interested  parties.  Please  contact  me  at  (202)  275-4841  if 
you  or  your  staff  have  any  questions  concerning  this  report.  Other  nuuor 
contributors  to  this  report  were  Jackie  B.  Guin,  Assistant  Director,  and 
Donald  F.  Lopes,  Evaluator-in-Charge. 


Sincerely  yours, 


Director,  Command,  Control,  Communications, 
and  Intelligence  Issues 
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Comments  From  the  Department  of  Defense 


Note:  GAO  comments 
supplementing  those  in  the 
report  text  appear  at  the 
end  of  this  appendix. 


See  comment  1 . 
See  comment  2. 
See  comment  3. 


See  comment  4. 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 
WASHINGTON.  DC  20301-3000 


ACQUISITION 

1  «  DEC  S3? 

Mr.  Frank  C.  Conahan 
Assistant  Comptroller  General 
National  Security  and  International 
Affairs  Division 
U.S.  General  Accounting  Office 
Washington,  DC  20548 

Dear  Mr.  Conahan: 

This  is  the  Department  of  Defense  (DoD)  response  to  the 
General  Accounting  Office  (GAO)  draft  report  entitled, 
"ELECTRONIC  WARFARE:  Laser  Warning  System  Production  Should  Be 
Limited"  (GAO  Code  395192/OSD  Case  9260).  The  Department 
partially  concurs  with  the  report  findings,  and  noncurs  with  the 
recommendation. 

The  GAO  concluded  that  the  redesigned  AVR-2  and  AVR-2A 
laser  warning  systems  being  procured  by  the  Army  are  unproven 
because  neither  system  has  been  subjected  to  operational  testing 
in  accordance  with  DoD  policy.  The  GAO  based  its  conclusion  on 
the  fact  that  operational  deficiencies  were  noted  during 
operational  testing  conducted  in  1985  on  the  initial  production 
AVR-2  system  and  that  reliability  had  not  been  assessed  during 
this  testing.  Additionally,  the  GAO  cocluded  that  the  redesign 
of  the  AVR-2  and  AVR-2 A  represented  a  material  change  to  the 
performance  of  the  system. 

In  reporting  the  1985  test  results,  however,  the 
independent  operational  test  organization,  D.S.  Army  Operational 
Test  and  Evaluation  Command,  concluded  that  the  AVR-2  was 
"operationally  suitable  and  increased  the  aircraft 
survivability."  Further,  the  Navy  operationally  tested  the 
system  in  1987,  and  also  concluded  that  the  system  was 
operationally  suitable  and  effective.  Additionally,  the  Army 
independent  operational  tester  stated  that  there  are  no 
outstanding  operational  issues  for  the  system,  after  concluding 
that  developmental  and  first  article  test  results  had  resolved 
outstanding  operational  deficiencies,  to  include  reliability. 
Subsequently,  the  Army  independent  operational  tester  concluded 
that  follow-on  operational  testing  of  the  AVR-2  system  was  not 
warranted.  That  position  was  supported  by  both  Army  and  Navy 
users,  as  well  as  the  Navy  independent  operational  tester. 
Regarding  the  "redesigned"  AVR-2  and  AVR-2A,  the  Army 
independent  operational  tester  and  the  user  concluded  that 
neither  of  those  system  changes  represented  a  material  change, 
in  hardware  or  software,  to  the  performance  of  the  systems  that 
would  warrant  follow-on  operational  testing. 
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See  comment  5. 
See  comment  4. 


See  comment  6. 


The  Department,  therefore,  disagrees  that  operational 
testing  on  the  initial  AVR-2  was  noncompliant  with  DoD  policy, 
or  that  the  redesign  of  the  AVR-2  and  AVR-2A  was  materially 
significant  so  as  to  warrant  follow-on  operational  testing.  The 
Department  does,  however,  agree  that  it  is  desirable  to  test  the 
integration  of  the  AVR-2A  on  the  main  aircraft  of  intended  use, 
the  AH-64  Apache  helicopter.  The  Army  plans  to  conduct  that 
testing  as  part  of  a  coordinated  test  and  evaluation  effort  with 
the  Army  independent  operational  tester  and  user.  The  Apache 
testing  is  scheduled  for  later  in  FY  1993. 

The  Department  also  does  not  agree  with  the  GAO 
recommendation  to  limit  AVR-2  and  AVR-2A  production,  based  on 
the  premise  that  inadequate  operational  testing  has  been 
conducted  or  is  planned.  Adequate  operational  testing  by  both 
the  Army  and  Navy  has  been  conducted  or  is  planned  to  support 
full-rate  production.  The  operational  needs  of  both  Services 
warrant  scheduled  introduction  of  the  AVR-2  and  AVR-2 A  on 
designated  aircraft. 


Sincerely, 


Director 

Tactical  Systems 
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Now  on  p.  3. 

See  comments  4  and  5. 


GAO  DRAFT  REPORT  -  DATED  NOVEMBER  10,  1992 
(GAO  CODE  395192)  OSD  CASE  9260 


"ELECTRONIC  WARFARE:  LASER  WARNING  SYSTEM 
PRODUCTION  SHOULD  BE  LIMITED" 

DEPARTMENT  OF  DEFENSE  COMMENTS 


FINDINGS 


•  FINDING  A:  Importance  of  Operational  Testing.  The  GAO 

concluded  that  the  performance  capability  of  the  redesigned 
AVR-2  and  AVR-2A  now  being  procured  is  unproven,  because 
neither  system  has  been  subjected  to  operational  tests. 

The  GAO  pointed  out  that,  contrary  to  DoD  policy,  the  Army 
does  not  plan  to  conduct  any  such  operational  tests.  The 
GAO  asserted  that  operational  testing  is  the  primary  means 
the  DoD  has  of  evaluating  weapon  system  performance  in  a 
combat  representative  environment — and  that  such  testing 
can  help  reduce  risks  in  acquisition  programs  by 
(1)  identifying  defective  systems  before  they  are  produced, 
and  (2)  verifying  the  correction  of  performance 
deficiencies.  The  GAO  explained  that  operational  testing 
is  distinguished  from  developmental  testing,  which  is  done 
to  verify  that  technical  performance  specifications  are  met 
and  to  determine  whether  a  system  is  ready  for  operational 
testing. 

The  GAO  contended  the  DoD  acquisition  policy  provides  that 
operational  testing  shall  be  structured  to  determine  the 
operational  effectiveness  and  suitability  of  a  system  under 
realistic  combat  conditions  and  to  determine  if  the  minimum 
acceptable  operational  performance  requirements  have  been 
satisfied.  The  GAO  further  contended  the  DoD  policy 
provides  that  a  system  may  not  enter  full-rate  production 
until  test  results  provide  reasonable  assurance  that  the 
design  is  operationally  acceptable.  In  addition,  the  GAO 
asserted  that  DoD  acquisition  policy  provides  that  one  of 
the  objectives  of  the  production  phase  is  to  conduct 
follow-on  operational  testing  to  confirm  system  performance 
and  verify  the  correction  of  deficiencies,  (pp.  4-5/GAO 
Draft  Report) 

POP  RESPONSE :  Partially  concur.  The  Department  disagrees 
that  the  performance  capability  of  the  "redesigned"  AVR-2 
and  AVR-2 A  is  unproven  in  operational  testing.  A 
production  representative  AVR-2  was  operationally  tested  in 
1985.  The  changes  incorporated  into  the  AVR-2  did  not 
materially  change  system  performance  and,  therefore,  in 
accordance  with  Department  of  Defense  Instruction  5000.2  do 
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3ee  comment  2. 
>ee  comment  7. 
see  comment  3. 


not  require  a  separate  operational  test.  Additionally,  the 
Navy  conducted  an  operational  test  of  the  AVR-2,  in  1987, 
and  found  the  system  to  be  both  operationally  effective  and 
suitable.  The  modifications  which  resulted  in  the  new  type 
classification  AVR-2A  improved  upon  the  performance 
previously  demonstrated.  The  addition  of  the  multiple 
integrated  laser  engagement  system  compatibility  is  a 
training  enhancement  to  the  system  and  does  not  represent 
an  operational  performance  capability.  The  user  currently 
has  no  outstanding  operational  issues  for  the  program. 

•  FINDING  B:  No  Operational  Teats  Planned.  The  GAO  reported 
that,  during  operational  tests  in  1985,  the  original  AVR-2 
failed  to  demonstrate  acceptable  performance.  The  GAO 
found  that  major  shortcomings  included  (1)  an  inability  to 
detect  multiple  threats,  (2)  an  inability  to  operate 
properly  with  helicopter  APR-39A  radar  warning  receivers, 
and  (3)  a  tendency  to  provide  false  warnings.  In  addition, 
the  GAO  observed  that  the  system  also  failed  to  demonstrate 
that  it  met  reliability  requirements.  Further,  the  GAO 
observed  that  the  system  configuration  used  in  testing 
weighed  22.2  pounds,  or  2.2  pounds  over  the  maximum 
allowable  weight. 

The  GAO  found  that,  following  the  198S  operational  tests  of 
the  AVR-2,  the  Army  made  design  changes  to  improve  system 
performance  and  reduce  weight.  The  GAO  reported  that  the 
changes  included  redesigning  the  system  receiver  and 
electronic  components  at  a  cost  of  $3.4  million,  or 
30  percent  of  the  system  original  development  cost  of 
$11.4  million.  The  GAO  further  reported  that  the  changes 
also  increased  unit  production  cost  by  $3,890.  The  GAO 
explained  that,  in  1991,  the  Army  began  a  series  of 
additional  design  changes  to  the  AVR-2  to  improve  the 
system  sensitivity,  and  thereby  allow  detection  of  certain 
threat  systems  at  greater  ranges.  In  addition,  the  GAO 
found  modifications  were  also  made  to  incorporate  a  device 
that  will  allow  its  use  with  a  training  system  for  Army 
pilots.  The  GAO  estimated  the  modifications  cost  about 
$8  million — or  70  percent  of  the  original  development  cost 
of  the  system — and  increased  unit  production  cost  by 
$9,061,  to  $134,312  each. 

The  GAO  contended  that,  despite  the  changes  in  system 
design  and  DoD  stated  policy,  the  Army  has  decided  that 
follow-on  operational  testing  is  not  necessary.  The  GAO 
noted  that,  consistent  with  DoD  policy,  the  Army  did  plan 
to  condu<  .  follow-on  operational  testing  in  1991 — using 
production  models  of  the  redesigned  AVR-2.  The  GAO 
observed  that,  according  to  the  test  plan,  several  critical 
issues  were  to  be  evaluated,  including  the  operational 
effectiveness  and  reliability  of  the  system.  The  GAO 
noted,  however,  that  the  follow-on  test  was  canceled 
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Now  on  p.  4 

See  comments  3  and  4. 


See  comments  3  and  5. 


because  of  OPERATION  DESERT  STORM.  (pp.  5-6/GAO  Draft 
Report) 

POD  RESPONSE:  Partially  concur.  The  user  has  determined 
that  there  are  no  outstanding  operational  issues.  The 
modifications  have  not  changed  the  operational  performance 
of  the  AVR-2/2A  system  so  as  to  warrant  additional 
operational  testing.  System  weight  reduction  was 
demonstrated  by  physically  weighing  the  system.  The 
operational  requirement  for  reliability  is  200  hours,  the 
contract  requirement  is  400  hours,  and  first  article 
testing  demonstrated  over  600  hours  reliability.  Technical 
testing  and  first  article  testing  results  were  accepted  by 
the  Army  Operational  Test  and  Evaluation  Command  in 
concluding  that  deficiencies,  noted  during  1985  operational 
testing,  had  been  corrected.  To  that  extent,  the  Army 
Operational  Test  and  Evaluation  Command  concluded  no 
operational  issues  remained  for  the  AVR-2.  Regarding  the 
AVR-2A,  the  Army  Operational  Test  and  Evaluation  Command 
recommended  that  a  follow-on  operational  test  be  conducted, 
only  if  open  issues  remain  that  cannot  be  satisfactorily 
addressed  in  other  testing.  Platform  specific,  follow-on 
operational  testing  is  ongoing  by  the  Navy  in  the  UH-1N 
helicopter  to  obtain  additional  data  on  the  AVR-2/APR-39 
compatibility  and  false  alarm  indications.  The  Army  will 
validate  the  integration  of  the  avr-2 A  on  the  AH-64  in 
testing  to  be  conducted  during  the  second  half  of  FY  1993. 

FINDING  C;  Substitute  Testing  is  Inadequate.  The  GAO 
found  that,  rather  than  conduct  follow-on  operational 
testing,  the  Army  is  relying  on  technical  tests  to 
demonstrate  system  performance.  The  GAO  concluded, 
however,  that  such  tests  are  not  a  valid  substitute  for 
operational  tests.  The  GAO  explained  that  technical 
testing  is  essentially  developmental  testing  and  is  not 
done  in  the  realistic  operational  environment  needed  to 
evaluate  system  operational  effectiveness  and  suitability. 
The  GAO  noted,  for  example,  that  tests  to  demonstrate  the 
AVR-2  reliability  were  held  in  the  contractor  plant  and 
were  not  done  under  the  cognizance  of  the  independent  Army 
operational  test  organization.  Moreover,  the  GAO  concluded 
that  DoD  policy  provides  for  demonstrating  system 
operational  effectiveness  and  suitability  in  operational 
tests — not  technical  tests. 

The  GAO  reported  that  its  other  work  has  shown  that, 
historically,  failure  to  conduct  operational  testing  to 
verify  the  adequacy  of  redesigned  systems  can  result  in 
adverse  consequences.  The  GAO  noted  that,  on  a  recent 
review  of  another  Army  electronic  warfare  system,  it  had 
found  that  design  changes  were  made  to  correct  a  serious 
shortcoming  discovered  during  operational  testing.  The  GAO 
further  noted,  however,  that  no  additional  operational 
testing  was  done  to  verify  the  adequacy  of  the  changes. 
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Now  on  pp.  4-5. 
See  comment  3. 


See  comment  2. 
See  comment  8. 


The  GAO  observed  that,  subsequently,  during  OPERATION 
DESERT  STORM,  the  system  proved  so  defective  that  Army 
pilots  stopped  using  it. 

In  addition,  the  GAO  observed  that  none  of  the  substitute 
testing  has  been  or  will  be  done  on  the  Army  Apache 
helicopter,  the  main  aircraft  slated  to  use  the  AVR-2A. 

The  GAO  noted  that,  during  a  recent  review  of  a  Navy 
electronic  warfare  system,  it  found  the  system  had 
successfully  passed  operational  testing  on  a  certain 
helicopter.  According  to  the  GAO,  based  on  those  tests, 
the  Navy  procured  several  hundred  systems  and  then 
operationally  tested  the  system  on  another  helicopter 
scheduled  to  use  the  system.  The  GAO  reported  that  the 
subsequent  tests  showed  that  the  system  degraded — rather 
than  enhanced — the  survivability  of  the  other  helicopter. 
Another  limitation  of  the  Army  substitute  testing  noted  by 
the  GAO  is  illustrated  by  the  AVR-2  electromagnetic 
compatibility  tests.  The  GAO  explained  that  the 
electromagnetic  compatibility  tests  refer  to  the  capability 
of  systems  to  operate  in  their  intended  environment, 
without  causing  or  suffering  from  interference  with  other 
systems.  The  GAO  pointed  out  the  AVR-2  electromagnetic 
compatibility  tests  were  conducted  on  a  Scout  helicopter 
that  had  much  of  its  electronic  equipment  either  missing  or 
inoperative.  In  addition,  the  GAO  emphasized,  the  Scout 
does  not  have  some  equipment  (such  as  the  ALQ-136  radar 
jammer)  that  is  used  on  the  Apache  and  Cobra  helicopters. 
The  GAO  concluded,  therefore,  that  the  Army  has  little 
assurance  of  the  AVR-2  electromagnetic  compatibility. 

(pp.  6-8/GAO  Draft  Report) 

POD  RESPONSE:  Partially  concur.  The  AVR-2  was 
operationally  tested  by  the  Army  in  1985  and  in  1987  by  the 
Navy.  In  reviewing  the  independent  evaluation  report  of 
the  operational  test  and  assessing  technical  and  first 
article  test  results,  the  Army  Operational  Test  and 
Evaluation  Command  determined  that  system  performance  had 
been  demonstrated  and  no  critical  operational  issues 
remained  that  warranted  follow-on  operational  testing. 

The  Navy  operational  test  determined  the  AVR-2  to  be 
operationally  suitable  and  effective.  The  Navy  is 
currently  conducting  operational  testing  of  the  AVR-2  on  a 
UH-1N  helicopter.  The  Army  plans  to  conduct  a  combined 
developmental  test/operational  test  of  the  AVR-2A 
integrated  on  the  AH-64  Apache  later  in  FY  1993. 
Participants  will  include  not  only  representatives  from  the 
Operational  Test  and  Evaluation  Command,  but  the  Army  user 
representative/combat  developer  from  the  Aviation  Center  at 
Fort  Rucker.  That  testing  will  include  bench,  ground  and 
flight  testing  and  will  verify  system  performance 
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Now  on  p.  5. 

See  comment  7. 


requirements,  including  electromagnetic  interference/ 
compatibility  and  weapon  firings.  Similar  testing  will  be 
conducted  on  each  Army  aircraft  designated  to  receive  the 
AVR-2A. 


RECOMMENDATION 

•  RECOMMENDATION :  The  GAO  recommended  that  the  Secretary  of 

Defense  require  that  production  of  the  AVR-2  and  AVR-2A  be 
minimized  until  operational  testing  provides  reasonable 
assurance  that  performance  is  satisfactory,  (p.  8/GAO 
Draft  Report) 

POD  RESPONSE!  Nonconcur.  Adequate  operational  testing  has 
been  conducted  or  is  planned  reasonably  to  assure 
operationally  suitable  and  effective  performance  of  the 
AVR-2  and  AVR-2 A  on  Army  and  Navy  aircraft.  It  continues 
to  be  the  Department's  position  that  production  of  the 
systems  should  proceed  at  the  full-rate. 


OMVN8IAD-M 
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The  following  are  gao’s  comments  on  the  Department  of  Defense’s  (dod) 
letter  dated  December  16, 1992,  and  its  accompanying  enclosure. 


GAO  Comments 


1.  dod’s  comment  omits  reference  to  the  performance  deficiencies 
detected  in  the  1985  tests,  including  the  inability  to  detect  multiple  threats 
and  operate  properly  with  the  helicopter’s  radar  warning  receiver  and  the 
tendency  to  provide  false  warnings.  In  addition,  the  AVR-2  failed  to 
demonstrate  that  it  met  requirements  for  reliability,  one  of  the  prime 
factors  in  determining  a  system’s  operational  suitability.  Because  system 
reliability  requirements  were  not  demonstrated,  conclusions  about  the 
AVR-2’s  operational  suitability  were  invalid. 

2.  The  Navy  tests  found  deficiencies  similar  to  those  revealed  in  the 
Army’s  1985  tests.  The  Navy  recommended  that  additional  operational 
testing  be  conducted  to  verify  that  performance  deficiencies  had  been 
corrected. 

3.  The  Army’s  operational  tester  initially  concluded  that  follow-on 
operational  testing  was  needed  to  resolve  performance  deficiencies,  but 
the  Army  canceled  the  tests  because  of  Operation  Desert  Storm. 
Subsequently,  based  on  the  technical  testing,  the  follow-on  operational 
tests  were  deemed  unnecessary.  As  discussed  on  page  5  of  our  report, 
these  technical  tests  were  not  a  valid  substitute  for  operational  tests. 

4.  As  indicated  on  page  4  of  our  report,  the  cost  of  modifying  the  AVR-2 
that  was  operationally  tested  in  1985  was  about  the  same  as  its  total 
original  development  cost.  The  modifications  were  made  to  correct 
deficiencies  and  improve  system  performance.  In  addition,  the  software 
for  the  AVR-2A  has  been  partially  rewritten  and  is  not  scheduled  to  be 
tested  until  February  1993.  Thus,  dod  cannot  be  certain  about  whether  the 
modifications  materially  changed  performance  until  the  redesigned  system 
is  retested. 

5.  dod  policy  clearly  provides  that  a  system’s  operational  effectiveness  and 
suitability  shall  be  determined  by  operational  testing,  dod  policy  does  not 
provide  for  evaluating  operational  performance  based  on  developmental 
testing,  as  was  done  on  the  redesigned  AVR-2  and  AVR-2A.  As  stated  on 
page  3  of  our  report,  developmental  testing  is  done  to  determine  whether  a 
system  is  ready  for  operational  testing. 
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6.  At  the  time  of  our  review,  operational  testing  of  the  AVR-2  had  revealed 
performance  deficiencies,  find  no  additional  operational  testing  was 
planned  to  verify  the  adequacy  of  design  changes.  Subsequently,  dod 
agreed  that  operational  testing  on  the  Apache  helicopter  was  needed. 
Despite  dod’s  disagreement  with  our  recommendation,  we  consider  dod’s 
action  to  be  partially  responsive  to  our  recommendation,  assuming  that 
the  additional  tests  are  accomplished  as  planned.  The  additional  tests 
should  reduce  acquisition  risks,  which  could  be  further  reduced  by  fully 
complying  with  our  recommendation. 

7.  Whether  the  modifications  improved  system  performance  has  not  been 
demonstrated  in  operational  testing. 

8.  These  tests  were  planned  in  response  to  the  draft  of  this  report. 


(MS1M) 
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